Fix Server v2 production issues (#767, #783, #787) #788
Conversation
|
@anivar please explain the consequences of the .args being loaded too late. What failure/issue does it cause? |
Fixes four critical production issues in llamafile Server v2: 1. **Fix .args loading timing** (llama.cpp main/main.cpp) - Move cosmo_args() call before determine_program() - Ensures --server --v2 flags in .args are seen when determining program mode - Fixes mozilla-ai#783 2. **Add URL prefix normalization** (llamafile/flags.cpp) - Consolidate consecutive slashes (//api/v1 → /api/v1) - Ensure leading slash, remove trailing slash - Validate AFTER normalization - Use static std::string for proper lifetime management (no memory leak) - Fixes mozilla-ai#767 3. **Robust partial write handling** (llamafile/server/client.cpp) - Implement full write loop to handle partial writes correctly - Handle EINTR (signal interruption) gracefully - Properly detect connection closure - Increase file transfer buffer from 512B to 16KB for better performance 4. **Remove aggressive client dropping** (llamafile/server/worker.cpp) - Remove code that kills oldest active connection when all workers busy - Let TCP listen backlog naturally queue incoming connections - Provides better UX (graceful queuing vs abrupt disconnection) - Fixes mozilla-ai#787 All fixes improve upon original PR mozilla-ai#788 with better error handling and no memory leaks.
anivar
force-pushed
the
fix-server-v2-production-issues
branch
from
4746d30 to
78a2261
Compare
|
@vlasky Great question! The timing issue causes a real production problem for anyone distributing llamafiles with embedded configuration. Here's what happens: when a user embeds This completely breaks the "distribute a self-contained llamafile" use case - you can't ship a llamafile that's pre-configured to run as a server via The fix is straightforward: load While I was in there, I also improved the other fixes - the URL normalization now avoids a memory leak by using static storage, the partial write handler does a proper retry loop instead of just one attempt, and the file transfer buffer got bumped to 16KB for better performance. |
2 participants
This PR fixes three critical issues preventing Server v2 from being used in production:
--url-prefix //pathcorrectly like the old server.argsbeing loaded too lateThe changes are minimal (~60 lines) and follow patterns from the existing server implementation.
Fixes #767, #783, #787